Goto

Collaborating Authors

 virtual world


Inside Fei-Fei Li's Plan to Build AI-Powered Virtual Worlds

TIME - Tech

Pillay is an editorial fellow at TIME. Pillay is an editorial fellow at TIME. Recent AI progress has followed a pattern. Across text, image, audio, and video, once the right technical foundations were discovered, it only took a few years for AI-generated outputs to go from merely passable to indistinguishable from human creation. Although it's early, recent advances suggest that virtual worlds--3D environments you can explore and interact with--could be next.


Google DeepMind is using Gemini to train agents inside Goat Simulator 3

MIT Technology Review

SIMA 2, which can figure out how to solve problems inside virtual worlds, could lead to more general-purpose agents and better robots. Google DeepMind has built a new video-game-playing agent called SIMA 2 that can navigate and solve problems in a wide range of 3D virtual worlds. The company claims it's a big step toward more general-purpose agents and better real-world robots. Google DeepMind first demoed SIMA (which stands for "scalable instructable multiworld agent") last year. But SIMA 2 has been built on top of Gemini, the firm's flagship large language model, which gives the agent a huge boost in capability. The researchers claim that SIMA 2 can carry out a range of more complex tasks inside virtual worlds, figure out how to solve certain challenges by itself, and chat with its users.


The Download: AI to measure pain, and how to deal with conspiracy theorists

MIT Technology Review

Researchers around the world are racing to turn pain--medicine's most subjective vital sign--into something a camera or sensor can score as reliably as blood pressure. The push has already produced PainChek--a smartphone app that scans people's faces for tiny muscle movements and uses artificial intelligence to output a pain score--which has been cleared by regulators on three continents and has logged more than 10 million pain assessments. Other startups are beginning to make similar inroads. The way we assess pain may finally be shifting, but when algorithms measure our suffering, does that change the way we treat it? This story is from the latest print issue of MIT Technology Review magazine, which is full of fascinating stories about our bodies. Someone I know became a conspiracy theorist seemingly overnight.


AI as intermediary in modern-day ritual: An immersive, interactive production of the roller disco musical Xanadu at UCLA

Winick, Mira, Agarwal, Naisha, Boussema, Chiheb, Lee, Ingrid, Vargas, Camilo, Burke, Jeff

arXiv.org Artificial Intelligence

Interfaces for contemporary large language, generative media, and perception AI models are often engineered for single user interaction. We investigate ritual as a design scaffold for developing collaborative, multi-user human-AI engagement. We consider the specific case of an immersive staging of the musical Xanadu performed at UCLA in Spring 2025. During a two-week run, over five hundred audience members contributed sketches and jazzercise moves that vision language models translated to virtual scenery elements and from choreographic prompts. This paper discusses four facets of interaction-as-ritual within the show: audience input as offerings that AI transforms into components of the ritual; performers as ritual guides, demonstrating how to interact with technology and sorting audience members into cohorts; AI systems as instruments "played" by the humans, in which sensing, generative components, and stagecraft create systems that can be mastered over time; and reciprocity of interaction, in which the show's AI machinery guides human behavior as well as being guided by humans, completing a human-AI feedback loop that visibly reshapes the virtual world. Ritual served as a frame for integrating linear narrative, character identity, music and interaction. The production explored how AI systems can support group creativity and play, addressing a critical gap in prevailing single user AI design paradigms.


Exploring the Potential of Citiverses for Regulatory Learning

Hupont, Isabelle, Ponti, Marisa, Schade, Sven

arXiv.org Artificial Intelligence

Citiverses hold the potential to support regulatory learning by offering immersive, virtual environments for experimenting with policy scenarios and technologies. This paper proposes a science-for-policy agenda to explore the potential of citiverses as experimentation spaces for regulatory learning, grounded in a consultation with a high-level panel of experts, including policymakers from the European Commission, national government science advisers and leading researchers in digital regulation and virtual worlds. It identifies key research areas, including scalability, real-time feedback, complexity modelling, cross-border collaboration, risk reduction, citizen participation, ethical considerations and the integration of emerging technologies. In addition, the paper analyses a set of experimental topics, spanning transportation, urban planning and the environment/climate crisis, that could be tested in citiverse platforms to advance regulatory learning in these areas. The proposed work is designed to inform future research for policy and emphasizes a responsible approach to developing and using citiverses. It prioritizes careful consideration of the ethical, economic, ecological and social dimensions of different regulations. The paper also explores essential preliminary steps necessary for integrating citiverses into the broader ecosystems of experimentation spaces, including test beds, living labs and regulatory sandboxes


Human Interaction for Collaborative Semantic SLAM using Extended Reality

Ribeiro, Laura, Shaheer, Muhammad, Fernandez-Cortizas, Miguel, Tourani, Ali, Voos, Holger, Sanchez-Lopez, Jose Luis

arXiv.org Artificial Intelligence

Abstract-- Semantic SLAM (Simultaneous Localization and Mapping) systems enrich robot maps with structural and semantic information, enabling robots to operate more effectively in complex environments. However, these systems struggle in real-world scenarios with occlusions, incomplete data, or ambiguous geometries, as they cannot fully leverage the higher-level spatial and semantic knowledge humans naturally apply. We introduce HICS-SLAM, a Human-in-the-Loop semantic SLAM framework that uses a shared extended reality environment for real-time collaboration. The system allows human operators to directly interact with and visualize the robot's 3D scene graph, and add high-level semantic concepts (e.g., rooms or structural entities) into the mapping process. We propose a graph-based semantic fusion methodology that integrates these human interventions with robot perception, enabling scalable collaboration for enhanced situational awareness. Experimental evaluations on real-world construction site datasets demonstrate improvements in room detection accuracy, map precision, and semantic completeness compared to automated baselines, demonstrating both the effectiveness of the approach and its potential for future extensions.


Meet your descendants – and your future self! A trip to Venice film festival's extended reality island

The Guardian

In the largest cinema at the Venice film festival, guests gather for the premiere of Frankenstein, Guillermo del Toro's lavish account of a man who dared to play God and created a monster. When the young scientist reanimates a dead body for his colleagues, some see it as a trick while others are outraged. "It's an abomination, an obscenity," shouts one hide-bound old timer, and his alarm is partly justified. Every technological breakthrough opens Pandora's box. You don't know what's going to crawl out or where it will then choose to go.


Agency, Affordances, and Enculturation of Augmentation Technologies

Duin, Ann Hill, Pedersen, Isabel

arXiv.org Artificial Intelligence

Augmentation technologies are undergoing a process of enculturation due to many factors, one being the rise of artificial intelligence (AI), or what the World Intellectual Property Organization (WIPO) terms the AI wave or AI boom. Chapter 3 focuses critical attention on the hyped assumption that sophisticated, emergent, and embodied augmentation technologies will improve lives, literacy, cultures, arts, economies, and social contexts. The chapter begins by discussing the problem of ambiguity with AI terminology, which it aids with a description of the WIPO Categorization of AI Technologies Scheme. It then draws on media and communication studies to explore concepts such as agents, agency, power, and agentive relationships between humans and robots. The chapter focuses on the development of non-human agents in industry as a critical factor in the rise of augmentation technologies. It looks at how marketing communication enculturates future users to adopt and adapt to the technology. Scholars are charting the significant ways that people are drawn further into commercial digital landscapes, such as the Metaverse concept, in post-internet society. It concludes by examining recent claims concerning the Metaverse and augmented reality.


SimPRIVE: a Simulation framework for Physical Robot Interaction with Virtual Environments

Nesti, Federico, D'Amico, Gianluca, Marinoni, Mauro, Buttazzo, Giorgio

arXiv.org Artificial Intelligence

--The use of machine learning in cyber-physical systems has attracted the interest of both industry and academia. However, no general solution has yet been found against the unpredictable behavior of neural networks and reinforcement learning agents. Nevertheless, the improvements of photo-realistic simulators have paved the way towards extensive testing of complex algorithms in different virtual scenarios, which would be expensive and dangerous to implement in the real world. PRIVE, a simulation framework for physical robot interaction with virtual environments, which operates as a vehicle-in-the-loop platform, rendering a virtual world while operating the vehicle in the real world. PRIVE, any physical mobile robot running on ROS 2 can easily be configured to move its digital twin in a virtual world built with the Unreal Engine 5 graphic engine, which can be populated with objects, people, or other vehicles with programmable behavior . PRIVE has been designed to accommodate custom or pre-built virtual worlds while being light-weight to contain execution times and allow fast rendering. Its main advantage lies in the possibility of testing complex algorithms on the full software and hardware stack while minimizing the risks and costs of a test campaign. The framework has been validated by testing a reinforcement learning agent trained for obstacle avoidance on an AgileX Scout Mini rover that navigates a virtual office environment where everyday objects and people are placed as obstacles. Recently, artificial intelligence (AI) models have achieved impressive performance in many applications, including cyber-physical systems (CPS).


Coupling Agent-Based Simulations and VR universes: the case of GAMA and Unity

Drogoul, Alexis, Taillandier, Patrick, Brugière, Arthur, Martinez, Louis, Sillano, Léon, Lesquoy, Baptiste, Nghi, Huynh Quang

arXiv.org Artificial Intelligence

Agent-based models (ABMs) and video games, including those taking advantage of virtual reality (VR), have undergone a remarkable parallel evolution, achieving impressive levels of complexity and sophistication. This paper argues that while ABMs prioritize scientific analysis and understanding and VR aims for immersive entertainment, they both simulate artificial worlds and can benefit from closer integration. Coupling both approaches indeed opens interesting possibilities for research and development in various fields, and in particular education, at the heart of the SIMPLE project, an EU-funded project on the development of digital tools for awareness raising on environmental issues. However, existing tools often present limitations, including technical complexity, limited functionalities, and lack of interoperability. To address these challenges, we introduce a novel framework for linking GAMA, a popular ABM platform, with Unity, a widely used game engine. This framework enables seamless data exchange, real-time visualization, and user interaction within VR environments, allowing researchers to leverage the strengths of both ABMs and VR for more impactful and engaging simulations. We demonstrate the capabilities of our framework through two prototypes built to highlight its potential in representing and interacting with complex socio-environmental system models. We conclude by emphasizing the importance of continued collaboration between the ABM and VR communities to develop robust, user-friendly tools, paving the way for a new era of collaborative research and immersive experiences in simulations.